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ABSTRACT 



Failure to consider errors of measurement when using 
partial correlation or analysis of covariance techniques can result 
in erroneous conclusions. Certain aspects of this problem are 
discussed and particular attention is given to issues raised in a 
recent article by Brewar, Campbell, and Crano. (Author) 
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ERRORS OF IREERENCE DUE TO ERRORS OF MEASUREMENT 
Robert L. Linn and Charles E, Werts 
Educational Testing Service 



Abstrac t 

Failure to consider errors of measurement when using partial correla- 
tion or analysis of covariance techniques can result in erroneous conclu- 
sions. Certain aspects of this problem are discussed and particular 
attention is given to issues raised in a recent article by Brewer^ Campbell^ 
and Crano. 




2 



ERRORS OF INFERENCE DUE TO ERRORS OF MEASURElvENT 



1,2 

Robert L. Linn and Charles E. Werts 
Educational Testing Service 

Brewer, Campbell, and Crano (I970) have justifiably criticized the use 
of partial correlation procedures in hypothesis testing research where errors 
of measurement are not taken into consideration. Ignoring measurement errors 
is much more serious when dealing with partial correlations than when dealing 
with simple zero-order correlations. In the latter case we know that the 
effect of errors of measurement, that are mutually uncorrelated and uncorre- 
lated with true scores, is to reduce the absolute value of the zero-order 
correlation between the fallible measures. As Lord (1965) has pointed out, 
however, we cannot ordinarily know the effect of such errors of measurement 
on a partial correlation. Errors of measurement can increase or decrease 
the magnitude of a partial correlation and may even result in a partial corre- 
lation of a different sign. 

As an alternative. Brewer et al. (1970) have suggested that factor 
analytic techniques be used to test a single-factor model before drawing 
conclusions about the nature of underlying conceptual variables. The pur- 
pose of the present paper is to reconsider the issues raised by these authors 
and the reasoning that led to their conclusions. Attention also will be 
given to some related arguments that were made in a recent attack on some 
commonly used methods for the evaluation of compensatory educational programs 
(Campbell & Erlebacher^ 1970)- Our thesis is that the basic problem is a 
lack of relevant information — a problem that cannot be resolved by the choice 
of a statistical procedure. 



O 

ERIC 



3 



- 2 - 



Relationship between Factor and Partial Correlation Analyses 

Ignoring errors of measurement; the relationship between the loadings on 
a single common factor and the partial correlations in the case of three 
variables is straightforward. The squared factor loadings on a single common 
factor can be expressed: 






( 1 ) 



for i;G;k = 1;2^3 ; i j ^ k ; where a^ is the factor loading on the 

single common factor for variable i and the p ' s are the intercorrelations 

2 

among the variables ; i;G;k . When p., - 0 ; a. is undefined. Assuming 

jk 1 

none of the three zero-order correlations equal zerO; the squared factor load- 
ing can be written as a function of the partial correlation; p., . : 

JK. 1 



a. = 1 - Cp., . ; 

1 ^ 



( 2 ) 



where 



C = 






2 

ik 






Provided that C is positive; it may be seen from (2) that when p . = 0 ; 

JK. 1 

2 2 
a. =1.0 and when p . < 0 ; a. > 1.0 . 

1 J K. . 1 1 • 

Frederic Lord (personal communication) suggested that the relationship 
between the factor and partial correlation analyses could be clarified by an 
example such as the one depicted in Figure 1. Given P^r v = *50 ; the 



possible values of P^. ^ and pu 

^ 1 ^ 3 ^ 2 ^ ^ 



are contained in the ellipse in Figure 1. 
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Regions of the figure that contain negative partial correlations are indicated. 
Factor loadings are denoted by a^ and regions that contain imaginary load- 
ings or squared loadings greater than 1.0 are indicated. 

Insert Figure 1 about here 



On line segment ^ 1 “ ^ a^ = 1.0 ; on line segment M 

^15 2 ~ ^ ^2 ^ ^ line segments aeb and cfd p ^2 j = 0 

a^ = 1.0 . Imaginary values of the a *s occur when one of the three zero- 
order correlations is negative while the other two are positive. 

Bias in Partial Correlation 

Brewer et al. ( 1970 ) argue that errors of measurement introduce a 
systematic bias into partial correlations. More specifically^ they state: 

”. . . the assumption is made that the variable being partialled out contains 
no unique components and is measured without error. Using partialling tech- 
niques when these assumptions are not met introduces systematic bias toward 
the unparsimonious conclusion that more conceptual factors are involved in a 
phenomenon than may actually be the case‘s (Brewer et al.^ 1970^ pp. 1-2), 
Although it is true that this may be the effect of a violation of the assump- 
tion of an error free measure ^ the bias may be in the opposite direction. It 
is easy to construct an example where the direction of the bias is toward a 
more parsimonious conclusion that fewer conceptual factors are involved in a 
phenomenon than is actually the case. Suppose ^ for example ^ that three latent 
variables ( ^ and ) had the following intercorrelations in the 

population ; 
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Pm ni - • 6 ^ 

1^12 

Pm rn = • 6 ^ 

and Pm m = *18 

igJ-j 

The correlation bet-weei^ and with partialed out is -.28125 

and the corresponding conclusion is that more than one conceptual variable 
is involved in this phenomenon. Suppose ^ however, that only a fallible 
measure of the first variable, say , was available, where 

and is uncorrelated with ^ *^2 "^3 ' Further, assume that 

the variance of is equal to twice the variance of (i.e., the 

reliability of X^ is . 50 ). Under these conditions the resulting intercor- 
relations among , T^ , and X^ would be : 

Py m = .6 /T 5 = .424 , 

= -6 /3 = .424 , 

Pm m = .18 
J-2-^3 

The correlation between and with X^ partialed out would be 0.0 

which would result in the more parsimonious, but’ erroneous conclusion that a 
second conceptual variable is not required. There is no intention to imply 
by this illustration that the bias of errors of measurement is typically, or 
even frequently, in the direction of producing a partial correlation that is 
closer to zero. Rather the point is that the direction of the bias cannot 
be determined without imposing additional assumptions (e.g., all reliabilities 
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and all zero-order and partial correlations among true scores are nonnegative ) 
arid/ or obtaining additional information such as the reliabilities of the 
measures. Given classical test theory assumptions, an estimate of the partial 
correlation among underlying true scores my be obtained by simply applying 
standard corrections for attenuation to the zero-order correlations. As Lord 
(1965) has noted, the need to make corrections for attenuation "...poses 
somewhat of a dilemma, since, first, it is often hard to obtain the particular 
kind of reliability coefficients that ai^e required for making the appropriate 
correction, and, further, the partial corrected for attenuation may be seri- 
ously affected by sampling errors. These obstacles can hardly justify the 
use of an uncorrected coefficient that may have the wrong sign, however" 

(Lord, 1965^ p. 56). 

The Single Factor Model vs. Partial Correlations 

As noted above. Brewer et al. (1970) have suggested that a single-factor 
model be tested before conclusions are drawn about the nature of underlying 
conceptual variables from partial correlations.. We shall argue that partial 
correlation analyses and factor analyses are based on different models and 
pose different questions. Knowing that a single factor can reproduce the 
intercorrelations among three observed fallible variables is not sufficient 
to draw conclusions about the partial correlations among the underlying con- 
ceptual variables or true scores that correspond to the observed scores. 

Assuming that three infallible measures ( ^ "^2 ^ ^ have a 

multivariate norml distribution, the partial correlation between Tp and 
Tj with partialed out has a very simple interpretation. It is equal 

to the zero-order correlation between Tp and . for any subpopulation 
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defined by a particular value of . ThuS; it provides a means of investi- 
gating the relationship between and with held constant in the 

above sense. The question of whether or not and are related when 

is held constant is not the same . as the question answered by a test for 
single factoredness for the observed scores- This is^ in principle^ acknowl- 
edged by Brewer et al- (I 970 ) in footnote number 3 where they discuss an 
example in which the control variable (l-Q-) has a factor loading of .43. 

They conclude that "--.if one has 'factored out' a variable upon which I.Q. 
loads only -43^ one has not in any meaningful sense 'factored out I.Q. 

(Brewer et alo ^ 1970; p. 7)- They go on to indicate that they are working 
on a technique of "focused factoring/' wherein the control variables are 
used to define the factor- Hopefully this procedure would exclude from the 
communality of a control variable only that variance that properly might be 
considered error variance - 

If the observed variables 0 ^^) related to their underlying true 

scores (T^) by the model; 

Xi = Ti + Ei ; i = 1;2;3 , 

where the errors (^j_) mutually uncorrelated and are uncorrelated with 

the true scores^ then (l) may be expressed in terms of the correlations among 

the true scores^ ^ ; and the reliabilities of the observed measures^ 

p. . , i-e-, the variance of T. divided by the variance of X. - Thus 

11 1 1 

„ ^T.T.Pt.T, 

2 1 n 1 k 

a. = p. . 

1 11 Prj, ^ 
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(5) 
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The correlation between T^. and with partialed out is proportional 

to 

^T.T, " ^T.T.^T.T, , 

j k 1 j Ik. 



which ^ given equation (5)^ equals: 



^ii 

Pt •p P'p •p I ^ 
i j i k \ 



- 1 
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Considering cases where a single factor reproduces the intercorrelations 

2 

among ^ ^2 ’ ^5 ^ ^ \ ^ ^ ^ above expression 

can be seen to have the following implications: 



When 


Pm r|i and. Pm ^ 

i j i k 


have the same 


Sign, 


1. 


2 

a. 

1 




implies 


.T, .T. ^ ° 
d k X 


3 


2 . 


2 

a. 

1 


= pii 


implie s 


.T, . T . "" ° 

d k X 


3 


3 . 


2 

a. 

1 


> p. . 
XX 


implies 


^T.T, .T. ° 

d k X 


3 


When 


Pr|i r|i and. ^ 

i j i k 


have opposite 


signs, 


1. 


2 

a. 

1 


<pii 


implies 


^T.T, .T. ° 

d k X 


3 


2 . 


2 

a. 

1 


” ^ii 


implies 


^T.T, .T. - 

d k X 


3 


3 . 


2 

a. 

1 


> p. . 
11 


implies 


^T.T .T. ^ ° 

d k X 


• 



These results show that when the correlations among the observed scores 
are reproduced by a single factor with squared loadings between 0 and 1^ no 
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conclusions are warranted regarding the partial correlations among the true ^ 
scores. Given positive reliabilities and nonzero intercorrelations among : 

i 

observed scores, if the three observed variables do not fit the single j 

I 

factor model, then the three partial correlations among true scores may be ! 

f 

positive or negative but not zero. I 

T. e relationship between the observed loadings on a single common factoif, 

t 

\ 

the partial correlations among observed scores, ana the partial correlations * 

t 

among true scores may be clarified by the example depicted in Figure 2. For» 

( 

the case ^ = .50 and = p^^ = P:z:z = *50 , Figure 2 shows the pos- ! 

1 2 i 

sible values of p^ ^ and p„ m - A set of regions is defined within 

13 2'^3 

which the factor loadings on a single common factor, the partial correlations 
among observed scores, and the partial correlations among true scores have 
specified characteristics. The ellipse in Figure 2 contains the values for; 



Insert Figure 2 about here 



which the determinant of the matrix containing the intercorrelations of , 

"^2 ’ "^3 greater than or equal to zero. Larger values of P>p ^ 

would define a thinner ellipse and smaller values a rounder ellipse. The 
mmibers inside the ellipse identify the various regions of the ellipse, and 
the letters identify line segments separating regions. For the regions in 
Figure 2, the factor loadings (a^) for a single common factor that will 
reproduce the intercorrelations among the observed scores, the partial correla- 
tions among the observed scores (p .) , and the partial correlations among 

JK.X 

the true scores (p ) are shown in Table 1. The values of a, , p,, . 
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and P<p ^ values of ^ and ^ on the boundaries between 

j k’ i ■^i G ■^i'^k 

regions are shorn in Table 2. 



Insert Tables 1 and 2 about here 



As was stated in implications A. 2 and B.2 above; P*]; ^ T 
2 k' i 

zero when a^ = p^^ . This occurs on line segments cO; cmd ; 

2 

and inj . When a^ = 1 (line segments to; and to) the partial; 

p_ . ; among observed scores is zero; however; p_ _ _ is nonzero. The 

jk.i Vk-^i 

location of line boh and line eok depends on the magnitude of p^^ and * 

boh is defined by points where 9m m = Pn -i Pm m Pm m and eok is defined by 
T^Tj 11 T2_T2 Tj^Tj ^ 

points where p = „ P„ m • A line where a, = 1 does not exist 

^15 ^12 ^ 2^3 ^ 

for this example because there are no possible values of p and p 

^13 ^2^3 

for which p equals p-.,p„ ^ P„ „ • Regions 2a, 3a, 4, 6a, 7a, and 8 

^12 -^ l -^3 -^ 2-^3 

are of interest since they define combinations of p^ ^ and p^ ^ for 
which a partial correlation for observed scores and a partial correlation for 
true scores have opposite signs. Regions 1; 2a; 5a; 4; 5; 6a; ^-nd 8 are 
where a satisfactory single-factor solution is obtained yet all three correla- 
tions between pairs of true scores with the third true score partialed out are 
nonzero. Different conclusions about the number of underlying conceptual 
variables involved in the phenomenon presumably would be drawn for instances 
in those regions. 

This problem should not be dealt with by simply invoking the principle 
of parsimony and thereby concluding that the fit of a single factor model 
indicates that there is only one dimension underlying the phenomenon. Rather; 
the problem should be dealt with by obtaining the additional information that 
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is necessary to make inferences -within a given model. A brief discussion of 
the use of multiple measures to obtain the needed information is presented 
below in the section on needed additional information. 

Errors of Measurement in the Analysis of Covariance 

Campbell and Erlebacher (1970) have provided a much needed criticism 
of the common misuse of the analysis of covariance as a means of trying to 
adjust for preexisting differences between experimental and control groups 
for the evaluation of compensatory education programs. They argue that 
"error” and "uniqueness" in the covariate result in bias when the groups 
differ on the direction of underestimating the slope of the regression of 
the dependent variable^ on the covariate (for a good discussion see Cochran^ 
1968). Porter (1967) has illustrated the nature of the resulting bias for 
various group differences in means on the covariate and on the dependent 
variable. When using the analysis of covariance; bias due to errors of 
measurement in the covariate might make a compensatory education program look 
bad (or good). 

The effect of "uniqueness" depends on its sources. If uniqueness is due 
to errors of validity (e.g.^ a perfectly reliable symptom of the underlying 
variable); then bias will result in the same way that it does from unreliabil- 
ity. On the other hand; if uniqueness merely refers to -unshared variance 
between the covariate and the dependent variable as in Campbell and Erlebacher *s 
( 1970 ) treatment of covariance adjustments; then the question of bias is 
ambiguous. Given independent errors; unshared variance may be due to vinrelia- 
bility; invalidity or a lack of perfect correlation be-tween underlying varia- 
bles. The latter is not a source of bias and should not be corrected for as 
is done by Campbell and Erlebacher ’s adjustment procedure. 

ERIC 
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This problem needs to be viewed from the perspective of Lord’s (1967) 
paradox. Lord has shovm that the comparison of preexisting groups by means 
of an analysis of covariance (statistician 2) and by means of an analysis of 
difference scores (statistician 1) can result in paradoxically different 
results; both of which are manifestly correct. In his hypothetical illustra- 
tive example; Lord depicted an experiment in which girls received one diet 
and boys another. For each group the mean and variance of the final weight 
was identical to the mean and variance of the initial weight. There were 
preexisting differences between the groups in mean weight; and for each 
group the within-group correlation between initial and final weight was .50. 
Assuming that the weight measures are error free; the above correlation 
would be the correlation between true initial weight and true final weight. 

In the absence of measurement errors the analysis of mean change would 
indicate no ’’treatment^' effect; whereas the analysis of covariance would 
indicate a “treatment*^ effect. 

Campbell and Erlebacher (1970) have suggested that in- pretest-posttest 
designs a ’’common-factor coefficient” might be used to correct for errors of 
measurement and uni^lueness in the covariate. Using the proper common factor 
coefficients for both pretest and posttest in the standard correction for 
attenuation formula would result in a "corrected” pretest-posttest correla- 
tion of 1.00. Assuming equal coefficients for the pretest and the posttest; 
the common factor coefficient for Lord’s example would be .50. Applying this 
"correction” would increase the slope of the within-group regression lines to 
1.00 and result in identical intercepts for the two groups. In essence; 
Campbell ahd Erlebacher have devised a roundabout way of siding with Lord’s 
first statistician. However; they have not resolved Lord’s paradox. Rather 
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than impose a restriction^ such as the one that the "corrected" correlation 
betvreen pretest and posttest be 1.00 (which^ in our opiniou; is unjustified)^ 
it would seem far better to conclude with Lord (1967) that "... there 
simply is no logical or statistical procedure that can be counted on to make 
proper allowances for uncontrolled preexisting differences between groups" 

(p. 305). 

Needed Additional Information for Fallible Measures 

Dealing with fallible measures will generally require additional assump- 
tions and additional information. In some instances ^ using parallel forms 
of one or more of the measures may provide the needed additional information. 

One difficulty with this procedm-e is that most observed measures are really 
symptoms or indirect measures of the variable or influence to be measured^ 
which is to say that even if the symptoms were measured with perfect reliabil- 
ity they would be imperfectly correlated with the "true" variable- The 
researcher must decide which symptoms are reflections of the relevant under- 
lying variable. This question is crucial since different sets of symptoms 
will typically define different "true" factors depending on the particular 
statistical procedure employed. The multitrait -multimethod approach intro- 
duced by Campbell and Fiske (1959) attempts to deal with this validity problem 
by using different methods of measuring the same variable. Correlations heti^een 
different method measures of the same trait typically will correlate less than 
equivalent measures^ i.e.^ in this model the classical psychometric approach 
using parallel forms is apt to underestimate correlations among underlying 
conceptual variables. An alternative way of stating this problem is to assume 
that part of the correlation between the two measures and X* of T^ is 
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due to correlated errors of measurement and that factors causing this correla- 
tion are uncorrelated with the true scores. In this case; the square root of 
the correlation between c?nd no longer provides a reasonable estimate 

of the correlation betvreen X^ and . Assuming that the errors are posi- 
tively correlated; the correlation bet\^een X^ and X* will overestimate 
the squared correlation between X^ and and using this inflated coeffi- 

cient to correct for attenuation wull result in the kind of under correction 
that Brewer et al. ( 1970 ) warned against. Correlated errors may; in fact; be 
one of the reasons that Brewer et al. wanted to correct for '^uniqueness.'' 

There are advantages; however; to formulating the problem in terms of corre- 
lated errors rather than simply saying that we should correct for uniqueness. 
The former makes it possible to devise procedures for estimating the desired 
coefficient (the correlation between X^ and T^ ) given the possibility of 
either positively or negatively correls,ted errors; whereas the latter only 
allows the conclusion that the correlation between X^^ and X* overestimates 
the desired coefficient if the errors are in fact positively correlated . 

Conclusion 

From our perspective; "focusing on the conceptual problem of choosing a 
one-factor vs. a two-factor model" (Brewer et al.; 1970; p. 3) distracts the 
researcher *s attention from the task of constructing a model which is consis- 
tent with everything we know or hypothesize about the phenomena under study. 
Any inferences will necessarily be no more valid than the assumptions made 
about reality. For heuristic purposes we have assumed that the linear addi- 
tive model was relevant; however; there is no rule of nature that effects are 
either linear or additive. No provision was made; e.g.; for catalytic; 
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feedback^ or interactional type influences. It is important for the research 
design to be set up to study the question of which of the plausible alternative 
models more closely simulates reality. Rather than focus on the conceptual 
problem of choosing a one -factor vs. a two- factor model; it seems to us far 
more worthwhile to spend time in designing the study to explore the relevant 
alternate models ; ensuring collection of the information necessary to test 
which is the best simulation of reality. Depending on the problem; the factor 
model may be one of the alternatives. The assumption that the factor model is 
a priori relevant appears to us to be unjustified given the current state of 
the art. 
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Footnotes 



^The research reported herein was performed pursuant to Grant No. 
OEG-2-700055 ( 509 ) with the United States Department of Healthy Education^ 
and Welfare and the Office of Education. 

^e are grateful to Frederic M. Lord for suggesting the idea that was 
used for the illustrative example in Figure 1. 
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Table 1 

Values of Factor Loadings and Partial Correlations 
for Regions of Figure 2 











Partial 


Correlations 




Partial Correlations 




Factor 


Loadings 


Among Observed Scores 




Among True 


Scores 


Region 


^1 


a^ 

2 


^5 


^ 23.1 


'^15.2 


^12.3 




.Ti ‘^Tj_Tj.T 


p 

1 ‘^rn m m 

2 ^1-^2-^p 


1 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


2a 


+ 




+ 


+ 


+ 


+ 


- 


+ 


+ 


2b 


a 

G 


+ 


+ 


- 


+ 


+ 


- 


+ 


4- 


5a 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


- 


4- 


5b 


+ 


G 


+ 


+ 


- 


+ 


+ 


- 


+ 


4 


+ 


+ 


+ 


+ 


+ 


+ 


+ 




- 


5 


- 


- 


+ 


- 


- 


+ 


- 


- 




6a 


- 


- 


+ 


- 


- 


+ 


+ 


- 


+ 


6b 


G 


- 


+ 


+ 


- 


+ 


+ 


- 


4- 


7a 


- 


- 


+ 


- 




+ 




+ 


+ 


7b 




G 


+ 


- 


+ 


+ 


- 


+ 


4- 


8 


- 


- 


+ 


- 


- 


■ + 


- 


- 


- 


9 


i 


i 


i 


- 


+ 


+ 


- 


+ 


4- 


10 


i 


i 


i 


+ 


- 


+ 


+ 


- 


4- 



8i 

G denotes that the factor loading is greater than 1.0 in absolute value. 
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Table 2 

Values of Factor Loadings and Partial Correlations 
for Lines Separating Regions in Figure 1 



Line 

Segment 


Factor Loadings 


Partial Correlations 
Among Observed Scores 


Partial Correlations 
Among True Scores 


^1 


a^ 

2 


^5 


^ 23.1 


^13. 2 


^12. 3 


^T2Tj.T^ ^Tj_T5.T 


'2 Vt2-T3 


ao 




0 


0 


- 


+ 




+ 


+ 


bo 


1 


+ 


+ 


0 


+ 


+ 


+ 


+ 


CO 




+ 


+ 


+ 


+ 


+ 


0 + 


+ 


do 


+ 


\/p22 


+ 


+ 


+ 


+ 


+ 0 


+ 


eo 


+ 


1 


+ 


+ 


0 


+ 


+ 


+ 


fo 


0 


U 


0 


+ 


- 


+ 


+ 


+ 


go 


u 


0 


0 


+ 


- 


+ 


+ 


+ 


ho 


-1 


- 


+ 


0 


- 


+ 


+ 


+ 


io 




- 


+ 


- 




+ 


0 


+ 


JO 


- 


"\/^22 


+ 


- 


- 


+ 


0 


+ 


ko 


- 


-1 


+ 


- 


0 


+ 


+ 


+ 


eo 


0 


U 


0 


- 


+ 


+ 


+ 


+ 


cmd 


+ 


+ 




+ 


+ 


+ 


+ + 


0 


i 


- 


- 




- 


-- 


+ 


- 


0 



^ denotes that the factor loading is undefined. 
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Figure Captions 



Fig. 1. Regions which define values of factor loading and partial 
correlations for possible values of given = .50 

^1 5 ^5 i 2 • 

Fig. 2, Regions which define values of factor loadings and partial 



correlations for possible values of m Prp given m 
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Pjj = *50 . 
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